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SPEC 



Method, and Apparatus for changing the output: delay of audio 
or video data encoding 

5 The invention relates to a method for changing the output 
delay of audio or video data encoding and to an apparatus 
for changing the output delay of audio or video data encod- 
ing . 

10 

Background 

Encoding systems with related video encoders and audio en- 
coders are used for various applications f e.g. for TV broad- 

15 casting purposes. In this case the video encoders can have 
variable encoding delay of up to 1-5 sec, depending for ex- 
ample upon the selected bit rate. Therefore, the audio and 
video delay should be aligned before multiplexing and trans- 
mitting the audio and video streams, because of buffer limi- 

20 tations in the decoders of consumer products. 

A basic mechanism for controlling delay within the audio en- 
coder by help of a time stamping mechanism can be found in 
the European patent application 99250009. In a multi-channel 

25 audio encoder board input time stamps are generated which 
become linked at least in one input processing stage with 
frames of audio data to be encoded, wherein the input time 
stamps or time stamps derived from the input time stamps re- 
main linked with the correspondingly processed frame data in 

30 the different processing stages in the processing, but axe 
at least in the last processing stage replaced by output 
time stamps. In each of theses stages the corresponding time 
stamp information linked with current frame data to be proc- 
essed is regarded in order to control the overall delay of 

35 the processing - 
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In order to allow switchable bit rates for the video and au 
.dio encoders on operators choice, for example for making 
space for an additional TV channel, a switchable delay of 
the audio encoder is desireable. However, the European pat- 
5 ent application 99250009 does not disclose, how such delay 
changes might be handled . 



Invention 

10 

It is one object of the invention to disclose a method for 
changing the output delay of audio or video data encoding, 
particularly for the purpose of switchable bit rates for the 
video and audio encoders on operators choice- This object is 
15 achieved by the method disclosed in claim 1. 

Tt is b further object of th© invention to disclose* an appa- 
ratus for changing the output delay of audio or video data 
encoding which utilises the inventive method. This object is 
20 achieved by the apparatus disclosed in claim 7. 

In principle, according to the inventive method input time 
stamps are generated which become linked with audio or video 
data to be 'encoded and are used to control the delay of the 

25 encoding process. Output time stamps are derived from the 

input time stamps by using a data delay constant and are as- 
signed to the encoded data for indicating the output time. 
The encoded data with assigned output time stamps are buff- 
ered before output, wherein for a change of the output delay 

30 the data delay constant is changed. Already assigned output 
time stamps remain unchanged. For data for which output time 
stamps are not already assigned, the output time stamps are 
calculated using the new data delay constant. 

35 Advantageous additional embodiments of the inventive method 
are disclosed in the respective dependent claims. 
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Drawings 

Embodiments of the invention are described with reference to 
the accompanying drawings/ which show in: 

5 

Fig. 1 a schematic flow chart of the method for changing 
the output delay; 

Fig* 2 a functional block diagram of a 4-channel audio en- 
coder using the inventive method. 

10 

Exemplary embodiments 
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15 Fig. 1 shows a schematic flow chart of the method for chang- 
ing the output delay. To audio samples ox audio coefficients 
an input time stamp information ITS is linked or assigned in 
the first method step 1. Then the audio samples or audio co- 
efficients are encoded in method step 2. In method step 3 it 

20 is checked, whether a change of the user-defined delay D is 
requested. If so, the processing equation for the output 
time stamps OTS = ITS + D is changed in method step 4. 
The input time stamp information ITS is then replaced by the 
output time stamps OTS in method step 5 before buffering in 

25 method step 6. The buffering is performed at the output of 
the encoding process, as the amount of memory required is 
lower at this side. Thus input data will typically be imme- 
diately encoded when received by the audio encoder, and be 
put after encoding into the delay buffer, especially in form 

30 of transport stream packets. 

Before sending the data to the output in method "step 9, the 
OTS are checked in method step 7. If a gap in the OTS conti- 
nuity occurs, stuffing data or zero data are inserted in 
35 method step 8. If on the other hand two packets with same or 
overlapping OTS are found in the delay buffer, this also re- 
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quires a special treatment in method step 8. One possibility 
is to discard packets indicating output times that are al- 
ready passed. One other possibility to handle this case is 
to write no further data into the output delay buffer begin- 
ning with the change request for the difference time, and to 
use the new delay time for the OTS calculation for all fol- 
lowing packets- Finally, the data are sent to the output in 
method step 9. 

In the following the treatment of the data r especially in 
method step 8, is described in more detail for an example 
delay increase or reduction, respectively, of 0.3 sec and an 
initial delay of 0,5 sec. 

For delay increase the delay buffer contains a certain 
amount of data equal to the presently effective delay time 
of 0.5 sec and this amount of data shall be increased to 0.8 
sec* This implies, that the output of encoded data from the 
delay buffer effectively needs to be stopped for 0.3 sec 
while data input to the delay buffer is continued. 

The data in the delay buffer are already encoded data repre- 
senting the continuation of the delay buffer already output 
just before. Therefore, the delay buffer is managed in the 
way, that after the change request for the delay time, the 
delay buffer continues to deliver data for 0.5 sec until all 
data that were in the delay buffer at the time of the delay 
time change are completely output- Then the output from the 
delay buffer is stopped, implying that either stuffing r 
zero data or no packets are sent to the transmitter/decoder 
chain* The stop then will last for a time of 0.3 sec, which 
is required to increase the delay buffer contents accord- 



This behaviour can be accomplished by use of the above men- 
tioned time stamp based delay control mechanism. All output 



ingly. 
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blocks, i.e. TS packets, that reside in the delay buffer at 
a given time are also stamped with an output time stainp that 
indicates the point in time when the packet should be sent 
out of the delay buffer and afterwards to the transmitter - 
Nothing needs to be changed for the packets that are already 
in the delay buffer, they are output as intended at their 
generation time. Immediately at the time where the audio de- 
lay D is changed, either directly or indirectly by operator, 
the processing equation for the output time stamps OTS = ITS 
+ D is changed, i.e. all OTS time stamps are increased by 
0.3 sec. All packets computed with the "old" delay are out- 
put orderly one after another by the output stage from the 
delay buffer. Then, after 0.5 sec, there will be a gap in 
the OTS continuity, i.e. the next packet will indicate an 
OTS that is indicating a time increased by 0.3 sec than the 
packet would have had without a delay change. The output 
stage then can send stuffing or zero data or even no pack- 
ets . 



20 Thus the net effect of delay increase for the user, i.e. the 
consumer listening at the decoder side, will be, that after 
the delay change request 

- the audio program does continue normal for the present ef- 
fective delay time of 0.5 sec, 

25 - the audio program shortly mutes for 0.3 sec, and 

- the program continues normally with the new delay of 0.8 
sec . 



On the operator side, upon delay change request the follow- 
30 ing happens 

- all program parts already input to the encoder will be se- 
quentially delivered to the user, 

- all audio program parts input after the delay switch will 
be separated by a short break from the previous parts on the 

35 user side. 



Printed:08-1 2-2000 



6 

The operator could use a program gap or the moment of a 
switch between distinct program parts for the delay time 
change in order to achieve a minimum irritation for the 
user - 

5 

Also at delay reduction the delay buffer contains a certain 
amount of data. The delay shall now be reduced from 0*5 sec 
to 0.2 sec delay* In this case, it will be necessary to 
normally continue the outputtiug process for 0.3 sec, while 
10 stopping the write of further input data into the delay 

buffer- Thus a short time of program material available at 
the input of the encoder will not be sent to the user. In 
principle the audio program at the user side could be conti- 
nous but a short piece would be cut out of the signal, 

15 

The delay reduction can be accomplished by the same time 
stamp based delay control. Immediately after the audio delay 
change request, the output time stamp OTS calculation is 
changed in a way, that the OTS indicate a point in time ear- 
20 lier by D.3 sec then ^without the delay change. 



If ,the writing of data packets into the delay buffer is con- 
tinued/ this would result in finding two packets with same 
or overlapping OTS in the delay buffer* As the packets are 

25 ordered in the delay buffer, output of the ^old" packets be- 
fore the audio delay change request would be continued nor- 
mally, until all data that were in the delay buffer at the 
time of change request are output. Then the next packets 
will indicate for 0.3 sec output times OTS that are already 

30 passed, so the output driver stage will have to discard 
these packets. 



Another method to handle this case is to write no further 
data into the output delay buffer beginning with the change 
35 request for the difference time of 0-3 sec, and to use the 
new delay time for the OTS calculation for all following 
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packets. In this case, the output stage of the encoder will 
find a more or less continous OTS sequence* 



10 



Thus the net effect of delay reduction for the user (con- 
sumer listening at the decoder side) will be, that after the 
delay change request 

- the audio program does continue normally for the present 
delay time of 0.5 sec, 

- then the program continues normally with the new delay of 
0.8 sec, but with a delay difference of 0.3 sec of audio 
program skipped. 



On the operator side, upon change request the following hap- 
pens 

15 - all program parts already input to the encoder will be ^se- 
quentially, i.e. normally delivered to the user, ; 

- the program material supplied to the encoder immediately 
after the change request for a following time period equal 
to the delay time difference of 0.3 sec will not be audible 

20 at the user side, 

- after this time period of 0*3 sec all audio program parts 
input to the encoder will be normally audible at the user 
side. 



25 Thus, the operator could use a program content switch to 
change delay time, and simply delay start of the next pro- 
gram by the delay time difference in order to guarantee 
that nothing of the program is lost for the listener. 

30 All discontinuities of the audio program, that become audi- 
ble to the listener, might be optionally softened by the 
encoder, i.e. proper fade in and out. Delay increase: fade 
out before the gap, fade in after the gap. Delay reduction: 
fade out before skipped part, fade in after that. In case of 

35 doing the delay changes together with audio program 
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switches, this might not be necessary, because the audic 
programmes will contain this switch. 



The inventive method can be used in an audio encoder as 
5 shown in Fig, 2. The encoder receives four stereo PCM input 
signals PCMA, PCMB, PCMC and PCMD. E.g. MPEG audio data are 
frame based, each frame containing 1152 mono or stereo sam- 
ples. The encoder operating system of Fig. 2 may include six 
DSPs (not depicted) for the encoding of the four MPEG chan- 
10 nels. These DSPs form a software encoder which includes the 
technical functions depicted in Fig. 2. A suitable type of 
DSP is for example ADSP 21060 or 21061 or 21062 of Analog 
Devices. As an alternative, the technical functions depicted 
in Fig, 2 can be realised in hardware. 

is 

Synchronisation of the software running on the six DSPs, or 
on corresponding hardware, is achieved using FIFO buffers 
wherein each buffer is assigned to one or some specific 
frames. This means that at a certain time instant a current 
20 frame as well as previous frames, the number of which de- 
pends from the quantity of available buffers, are present in 
the processing stages. 

Between some of the stages asynchronous buffers ASBUF are 
25 inserted which allow asynchronous write and read operations. 
Between other stages synchronous buffers EOF are sufficient. 
The PCM input signals PCMA, PCMB, PCMC and PCMD each pass 
via an asynchronous buffer to a respective converter CONA, 
CONB, CONC and COND. In such converter an integer-to- 
30 floating representation conversion of the audio samples to 
be encoded may take place. It is also possible that the en- 
coder processes integer representation audio samples. 
In such converter also one. or more kinds of energy levels in 
a frame may be calculated, e-g, energy of all samples of the 
35 frame or average energy of the samples of a frame. These en- 
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ergy values may be used in the subsequent psychoacoustic 
processing. 




In addition, in such converter the possibly adapted encoding 
5 parameters can become linked with the frame audio data. In 
respective parameter encoders PENCA, PENCB, PENCC and PENCD 
the original encoding parameters may be converted as de- 
scribed above and then fed to CONA, CONB, CONC and COND, re- 
spectively • 

10 

Via asynchronous buffers the output data of CONA, CONB, CONC 
and COND are fed in parallel to subband filters SUBA, SUBB, 
SUBC and SUBD and to first left and right channel psycho- 
acoustic calculators PsycholA^L, PsycholA_R r FsycholB_L, 

15 PsycholB_R, PsycholC_L, PsycholC_R, PsycholD_L and Psy- 

cholD_R, respectively- The subband filters divide the total 
audio spectrum into frequency bands, possibly using FFT, and 
may calculate the maximum or scale factor of the' coeffi- 
cients in a frequency band or subband. Within the frequency 

20 bands a normalisation may be carried out. The subband fil- 
ters take into account the above time stamp information and 
possibly the relevant encoding parameters read from the cor- 
responding upstream asynchronous buffer, 

25 The first psychoacoustic calculators perform an FFT having a 
length of e.g. 1024 samples and determine the current mask- 
ing information. Each first psychoacoustic calculator can be 
followed by a second psychoacoustic calculator Psycho2A_L, 
Psycho2A_R, Psycho2B_L, Psycho2B_R, Psycho2C_JL, Psycho2C_R, 

30 Psycho2D_L and Psycho2D_R r respectively , which evaluates the 
maximum or scale factor values previously calculated in the 
subband filters- The first and second psychoacoustic calcu- 
lators take into account the above time stamp information 

-ftrNH jnnrni V>1 y -treO *-»<^ A-rtoorling j* m.i* slim.* * *#«-«b«L £a* alw — 

35 responding upstream asynchronous buffers . 



The output signals of Psycho2A_L, Psycho2A_R, Psycho2B_L, 
Psycho2B_R, Psycho2C_L, Psycho2C_R, Psycho2D_L and Psy- 

ake^D^n Aa*-e used in J=>i"b ^HoCwa. u unU ^uauLlscj.o Bal/^/'C Ji / 

Bal/Q/EJB, Bal/Q/EJZ and Bal/Q/E_D, respectively, for deter- 
5 mining the number of bits allocated and the quantisation the 
audio data coefficients coming from the associated subband 
filter via a buffer. It is also possible to calculate in the 
second psychoacoustic calculators in addition what is being 
C^loulatQd in the firot poyohoacouaftift An 1 dilators and 
10 thereby to omit the first psychoacoustic calculators. 

Finally, the outputs of Bal/Q/E_A, Bal/Q/E_B, Bal/Q/E_C and 
Bal/Q/E_D pass through an asynchronous buffers and output 
interfaces AES-EBU_A, AES-EBU_B, AES-EBU_C, AES-EBU_D, re- 
15 spectivcly, wliioh d«liv*i" the eri^vuUsi aLcicu uuLpuL £>ly udls 
PCM_0ut_A, PCM_Out_B, PCM_0ut_C, PCM_0utJD, respectively. 
The output interfaces may correspond to IEC 958. 

A video encoder includes the following stages: block differ- 
20 ence stage, DCT (discrete cosine transform) , quantisation 
and in the feedback loop inverse quantisation, inverse DOT, 
motion compensated interpolation the output of which is in- 
put to the block difference stage, wherein the output of the 
quantisation is possibly VLC (variable length coding) en- 
25 coded and buffered before final ontmii- and the hnff^r -Pill- 
ing level is us^d to control the quantisation in ouch a way 
that encoding artefacts are masked as far as possible. 

in Liiia micuu^i Llie ruiiuwing elements are required: 
30 • a system time base supplying a system time, that delivers 

unique values for all comprised input and output stages; 
♦ hardware and/or software mechanisms that relates system 

time base and input data to obtain sufficiently precise 

input timo ctompo (ITS) } 

35 ♦ hardware and/or software mechanisms that relates the sys- 
tem time base with the data output to obtain sufficiently 
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precise output according to the output time stamps (OTS) . 



These elements are used in the following way: 

a) Per input interface of the system, input data are re- 
lated to the system timer, i.e. an input time stamp ITS 
along with the incoming data is obtained and assigned to 
the data frames. For example , the system time of the 
sampling moment of the first sample of a sampled audio 
data block or frame is used therefore. 

b) In the case of multiple input s, the input data blocks 
can be realigned across the channels by the time infor- 
mation given by the input time stamps. 

Example 1: multichannel audio inputs distributed over 
several two-channel interfaces. 

Example 2; bitstream outputs of multiple stereo encoders 
shall be multiplexed into an MPEG TS (transport stream) 
with a well-defined time relation across the channels, 
i.e. equal delay possibility. 

c) From input time stamps ITS and the intended overall de- 
lay D output time stamps OTS for the output data are 
calculated. 

In the easiest processing case where one output data 
block per input data block is calculated by the system, 
the intended output time of each output block is given 
by OTS(n)= ITS (a) + D , n = 0,1,2... denoting the data 
block number. 

In the case where per input data block several output 
data blocks or even a non-integer number of output 
blocks is to be generated, the OTS for each of the out- 
put blocks can be interpolated according to the corre- 
sponding time ratios. An example is the MPEG encoder 
1152 samples input, or an MPEG PES packet with one or 
more ESPs (elementary stream packets), and the MPEG TS 
transport stream wherein the packets have a length of 
138 Bytes, i.e. for each sample frame 3 to 7 TS packets 
are required for transmission. 



*-* V W V V* ' 
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12 

d) Each output interface examines the output data blocks , 
as supplied to the output buffer by the processing 
stages, for their related GTS in relation to the present 
system time as described in detail in the above descrip- 
tion of figure 1> If OTS is indicating a time instant 
that has already passed the output data block can be 
discarded or can be output immediately, depending on the 
application. If the OTS is pointing to a future time in- 
stant the output stage will wait until that time instant 
has come and during the wait time can either output 
nothing or a defined filling pattern. 



A couple of related mechanisms are available which can be 
used in different combinations as required for the I/O proc- 

15 ess to be implemented. 

In an example minimum hardware scenario the system makes use 
of a single hardware timer, which is typically already part 
of each DSP, in combination with some kind of regular or 
controlled output driver software activation. The re$t of 

20 the delay control is then executed by the DSP. In principle 
two timer functions are required: 

• A 'getTimeO 1 function, that allows the software to ask 
for the actual system time. Upon reception (DMA or INT 
based) of the beginning or the end of each input data 

25 block the getTimeO function can be used to obtain the ITS 
for that block. 

• The output function may need a certain delay before send- 
ing an output block for which the corresponding processing 
has finished. This can either be done in a polling manner, 

30 i.e. by cyclic control of the actual system time vs. OTS 

if some kind of cyclic activation is available, or by a 
specific timer-based delay function that issues an inter- 
rupt after a definable delay ' interruptAf ter (DELAY) ? or 
' threadActivityAfter (DELAY) 1 function* 

35 The single hardware timer, operating as a backward counter 
with interrupt at zero, and the input and output DMA block- 
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complete interrupts which are build-in functions of the 
DSPs, are used. 

The single hardware timer can provide the functions f inter- 
rupt After (DELAY) ' and T getTime ( } ' , wherein for the latter 
function the subsequent delay times loaded to the timer are 
summed up to obtain a continues system time and wherein sev- 
eral ' int err up tAfter (DELAY) 1 functions can run in parallel- 



In the case of a multi-DSP system with each DSP implementing 

io it's own timer, but distributing inputs and outputs with de- 
lay requirements between th^m across different DSPs, there 
is a problem of timer synchronicity • This problem can be 
solved by a special cyclic interrupt signal (e.g. of 10ms 
length) that is applied to all DSPs in the system and that 

15 is used to resynchronise the system times. The counter out- 
put word may have the format . iiii. ffff, wherein iiii is in- 
terpreted as integer part and ffff is interpreted as frac- 
tional part. Every 10ms iiii is incremented by f l'. This 
event is transmitted to the DSPs and is counted therein. The 

20 maximum manageable value for DELAY depends on the word 

length of iiii. Thereby the interrupt indicates the moment 
of resynchronisation and during the interrupt period a mas- 
ter value iiii for the time is transmitted from one master 
DSP to all other DSPs. At this time instant ffff is set to 

25 2ero. 



While the delay control described in above example scenario 
requires minimum hardware only and thus leaves most of the 
job to (cheaper and more flexible) software, the disadvan- 
30 tage is that the delay time accuracy is limited by e.g. in- 

4*^-irT-iip+- 1. ^tonoy -timoo, m.a.ia*.MA**** fcn%*wvvpi a!=» X <z* IJLmww ^n«J. , 

in r?<;p of a multi-DSP system, bus arbitration dolayc. 
Tf t"hf? achieved accuracy i* not sufficient, hardwaro on- 
hduueuieiiLs by used that make the process o£ ontainmg 

35 ITS relative to system time and output L aL OT3 icla Liv« Lu 

system time more accurate . a comomea solution with software 
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d*t£ViVij.uiizy Lht; luugli point in time and special Hardware es- 
tablishing thfi exact point in time will achieve a compromise 
between required DSP reaction time (tends to be slow com- 
pared to hardware) and hardware complexity (tends to be more 
5 complex the longer the time periods are) . 

A theoretical overflow beginning at the buffers related to 
the last processing stages and proceeding to the. buffers re- 
lated to the front processing stages is prevented by hand- 
10 shake between the DSPs. 

pli^d £<t 1? ^itVg-lo DOD ayatwris Aiid. -Co-u cni_y wLli^i. Kind ^-C j,<=:al 

time processing. 

15 

In cimplifiad applications, e.g. in an AC 3 dscodfti^ comput- 

constraint that, fchfc prnnRRRlng rim* fnr *srh block must b<a 
shorter than the frame period is an often used approach* 

tributed processina innlnriino spHftina inrn <=ip™»r;=il ^nhqp- 

ubj^s. m tnis case eacn processing stage can be forced into 
a well defined 'time slot 1 , whftre -Pot panh prnna^^ina <*tP>o& 
75 tn© proc^wwmg tim« t muct loo cfrox-fcox tJncm tho -fc imo plot 

length. In contrast to the solution described first, instead 
straint per time slot/processing stage. 

with diffarant oncrtriing paramo.t^ro bt^auoc MrEC allows «.q, 
various sample frequencies and overall data rates. 

Tlits iiiv^iiLiv-ui hc*» Lhe eidv^iitaye Lhat the shortest disruption 
33 jjcxIoJ c:ii»uj.cd, when cm upyxdioi changes ihe aexay. 
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The invention can be used to ensure a fixed but variable en- 
coding delay for any variable bit rate encoder or encoder 
which need a variable processing time. 



5 The invention can especially be used for MPEG 1, 2 and 4 Au- 
dio encoding and decoding for MPEG layers 1, 2 or 3, Digital 
Video Broadcast DVB, for AC-3, MD and AAC processing, for 
DVD processing and Internet applications concerning audio 
data encoding and decoding. 
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Claims 



1. Method for changing the output delay of audio or video 
data encoding, wherein input time stamps are generated 
which become linked with audio or video data to be en- 
coded and are used to control the delay of the encoding 
process, wherein output time stamps are derived from the 
input time stamps by using a data delay constant and are 
assigned to the encoded data for indicating the output 
time/ and wherein the encoded data with assigned output 
time stamps are buffered before output, characterised in 
that for a change of the output delay said data delay 
constant is changed, that already assigned output time 
stamps remain unchanged and that for data for which out- 
put time stamps are not already assigned, the output time 
stamps are calculated using the new data delay constant. 

2. Method according to claim 1, wherein during a gap in the 
sequence of output time stamps appearing at delay in- 
crease , stuffing data or zero data or even no data are 
sent . 

3* Method according feo claim 1 or 2, wherein, for Jala wiLh 
same or overlapping output time stamps appearing at delay 
reduction, tho l^tor da-ta indicating output time a tamp a 
that are already passed are discarded. 

4. Method according to any of claims 1 to 2, wherein for de- 
lay reduction no further data are written into the output 
delay buffer beginning with the delay change request for 
a time duration corresponding to the difference between 
the old data delay constant and the new data delay con- 
stant, and wherein the new data delay constant is used 
for the calculation of the output time stamps for all 
following data. 
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5. Method according to any of claims 1 to 4, wherein discon- 
tinuities of the audio or video data are softened by the 
encoder . 



5 6* Method according to any of claims 1 to 5/ wherein the 

discontinuities are softened by fading in before the gap 
or skipped part and fading out after the gap or skipped 
part • 



10 7 . Apparatus for performing a method according to any of 
claims 1 to 6. 
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Abstract 



For TV broadcasting purposes encoding systems with 
related video encoders and audio encoders are used. The au- 
5 dio and video delay die aligned before multiplexing and 
transmitting the audio and video streams- According to a 
time stamping mechanism input time stamps are generated 
which become linked with data to be encoded, and are re- 
placed before output by output time stamps, which are de- 

10 rived from the input time stamps by using a data delay con- 
stant* The input time stamps are used to control the delay 
of the encoding process and the output time stamps are indi- 
cating the output time. In order to allow a switchable out- 
put delay the data delay constant can be changed. Already 

15 assigned output time stamps remained unchanged. For data for 
which output time stamps are not already assigned, the out- 
put time stamps are calculated using the new delay constant . 



Fig. 1 
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